Say it as you mean it - analyzing free user comments in the VOICE awards corpus

Usability questionnaires usually contain scales related to effectiveness, efficiency and overall satisfaction which provide a quantitative value for the user’s opinion. However, analyzing quantitative data often does not show the reason underlying for a good or bad opinion. Simple questions like “What did you like about the system?” and “What did you not like about the system?” can shade light on the underlying reasons, but a lot of effort is needed for the analysis of such data. Nevertheless, the answers to these questions contain the users’ opinion in their own words and hence often show high correlation with the overall rating of the system. In the frame of the SpeechEval [1] project we analyzed the German VOICE Awards corpus over three consecutive years, categorizing the answers to these two free text questions and analyzing correlations between the categories and the overall rating of the systems. We used the data to build a general linear model for predicting the overall rating.